---
title: Heroic Query Language
---
<h2>{{ page.title }}</h2>

<p>
  Heroic uses a JSON-based language to define queries.
  This can also be expressed using an experimental DSL called HQL
  (Heroic Query Language).
</p>

<p>
  Queries have the following structure.
</p>

<pre><code class="language-hql">
&lt;aggregation&gt;
  [from &lt;source&gt;]
  [where &lt;filter&gt;]
  [as &lt;key&gt;=&lt;value&gt;[,&lt;key&gt;=&lt;value&gt;]]
  [with &lt;key&gt;=&lt;value&gt;[,&lt;key&gt;=&lt;value&gt;]];
</code></pre>

<p>
  References have the following structures.
</p>

<pre><code class="language-hql">
let $&lt;name&gt; = &lt;query&gt;;
</code></pre>

<p>
  <em>Complex Queries</em> are queries referencing other queries through a
  reference.
  The following is an example of this
</p>

<pre><code class="language-hql">
let $a = average by host | sum by site;

$a / shift($a, -7d) - 1.0
  where what = cpu-usage and role = heroic;
</code></pre>

<p>
  The following is a complete example of a HQL-based query:
</p>

<pre><code class="language-hql">
average by host | sum by site
  from points(1d)
  where role=heroic and what=cpu-idle
  with size=5m
</code></pre>

<h3 id="json-dsl">
  JSON vs HQL
  <a class="link-to" href="#json-dsl"><span class="glyphicon glyphicon-link"></span></a>
</h3>

<p>
  JSON is typically used when a query is built programatically because the structure is unambigious in terms of precedence and escaping.
  There is also a ton of language support for it, and it meshes well with restful APIs.
</p>

<p>
  The HQL was developed to make it easier for humans to express queries or filters in a manner which is more convenient.
  The language is infix, and simple strings do not have to be escaped (e.g. <code class="language-hql">host</code> vs. <code class="language-hql">&quot;host&quot;</code>)
</p>

<p>
  A primary Goal of the HQL is that it should act as a complement to the JSON queries.
  Any query can be expressed either in JSON or HQL.
</p>

<p>
  The following is an example filter expressed both in a JSON, and in the HQL.
</p>

<pre><code class="language-hql">
$key = "hello kitty" and host = foo.example.com
</code><code class="language-json">
["and", ["$key", "hello kitty"], ["=", "host", "foo.example.com"]]
</code></pre>

<p>
  To test this principle you can fire up the <a href="shell">Heroic Shell</a> and run the following commands:
</p>

<pre><code class="language-bash">
$ tools/heroic-shell

heroic> parse-query --no-indent "average by host | sum by site"
{"aggregation":{"type":"chain","chain":[{"type":"group","of":["host"],"each":{"type":"sum"}},{"type":"average"}]}}
</code></pre>

<h3 id="aggregations">
  Aggregations
  <a class="link-to" href="#aggregations"><span class="glyphicon glyphicon-link"></span></a>
</h3>

<p>
  Main Article: <a href="aggregations">Aggregations</a>
</p>

<p>
  Aggregations are expressed as function calls, or typed documents. Aggregations can be done on both tags and resource identifiers.
  See the following example using the <a href="aggregations#sum">sum</a> aggregation.
</p>

<div class="table-responsive">
<table class="table table-bordered">
  <tr>
    <td>
      <pre><code class="language-hql">
      sum(3m)
      </code></pre>
    </td>
    <td>
      <pre><code class="language-json">
      {"type": "sum", "size": "3m"}
      </code></pre>
    </td>
  </tr>
</table>
</div>

<h3 id="filters">
  Filters
  <a class="link-to" href="#filters"><span class="glyphicon glyphicon-link"></span></a>
</h3>

<p>
  A filter reduces the number of selected time series, if no filter (or the true filter) is used, then it is implied that <em>all</em> time series in the database is selected.
</p>

<img style="width: 100%;" src="{{ 'images/filter-dsl.svg' | relative_url}}"></img>

<h3>Available Filters</h3>

<h4>Boolean Operators</h4>

<div class="table-responsive">
<table class="table table-bordered">
  <tr>
    <th></th>
    <th>Description</th>
    <th>Parsed</th>
    <th>JSON</th>
  </tr>

  <tr>
    <th scope="row">And</th>
    <td>Matches if all child statements match.<code class="language-hql">true</code> if empty.</td>
    <td><pre><code class="language-hql">&lt;a&gt; and &lt;b&gt;</code></pre></td>
    <td><pre><code class="language-json">["and", &lt;a&gt;, &lt;b&gt;]</code></pre></td>
  </tr>

  <tr>
    <th scope="row">Or</th>
    <td>Matches if any child statements match.<code class="language-hql">false</code> if empty.</td>
    <td><pre><code class="language-hql">&lt;a&gt; or &lt;b&gt;</code></pre></td>
    <td><pre><code class="language-json">["or", &lt;a&gt;, &lt;b&gt;]</code></pre></td>
  </tr>

  <tr>
    <th scope="row">Not</th>
    <td>Matches if child statement does not match.</td>
    <td><pre><code class="language-hql">!&lt;a&gt;</code></pre></td>
    <td><pre><code class="language-json">["not", &lt;a&gt;]</code></pre></td>
  </tr>

  <tr>
    <th scope="row">Override Grouping</th>
    <td>Override grouping.</td>
    <td><pre><code class="language-hql">(&lt;a&gt; and &lt;b&gt;) and &lt;c&gt;</code></pre></td>
    <td><pre><code class="language-json">["and", ["and", &lt;a&gt;, &lt;b&gt;], &lt;c&gt;]</code></pre></td>
  </tr>
</table>
</div>

<h4>Operators</h4>

<div class="table-responsive">
<table class="table table-bordered">
  <tr>
    <th></th>
    <th>Description</th>
    <th>Parsed</th>
    <th>JSON</th>
  </tr>

  <tr>
    <th scope="row">Tag Matches</th>
    <td>Tag foo equals to bar.</td>
    <td><pre><code class="language-hql">&lt;foo&gt; = &lt;bar&gt;</code></pre></td>
    <td><pre><code class="language-json">["=", &lt;foo&gt;, &lt;bar&gt;]</code></pre></td>
  </tr>

  <tr>
    <th scope="row">Tag Does Not Match</th>
    <td>Tag foo does not equal to bar.</td>
    <td><pre><code class="language-hql">&lt;foo&gt; != &lt;bar&gt;</code></pre></td>
    <td><pre><code class="language-json">["not", ["=", &lt;foo&gt;, &lt;bar&gt;]]</code></pre></td>
  </tr>

  <tr>
    <th scope="row">Tag In</th>
    <td>Tag foo equals to either bar or baz.</td>
    <td><pre><code class="language-hql">&lt;foo&gt; in [&lt;bar&gt;, &lt;baz&gt;]</code></pre></td>
    <td><pre><code class="language-json">["or", ["=", &lt;foo&gt;, &lt;bar&gt;], ["=", &lt;foo&gt;, &lt;baz&gt;]]</code></pre></td>
  </tr>

  <tr>
    <th scope="row">Tag Not In</th>
    <td>Tag foo does not equal to either bar or baz.</td>
    <td><pre><code class="language-hql">&lt;foo&gt; not in [&lt;bar&gt;, &lt;baz&gt;]</code></pre></td>
    <td><pre><code class="language-json">["not", ["or", ["=", &lt;foo&gt;, &lt;bar&gt;], ["=", &lt;foo&gt;, &lt;baz&gt;]]]</code></pre></td>
  </tr>

  <tr>
    <th scope="row">Tag Exists</th>
    <td>Tag foo exists.</td>
    <td><pre><code class="language-hql">+&lt;foo&gt;</code></pre></td>
    <td><pre><code class="language-json">["+", &lt;foo&gt;]</code></pre></td>
  </tr>

  <tr>
    <th scope="row">Tag Does Not Exists</th>
    <td>Tag foo does not exist.</td>
    <td><pre><code class="language-hql">!+&lt;foo&gt;</code></pre></td>
    <td><pre><code class="language-json">["not", ["+", &lt;foo&gt;]]</code></pre></td>
  </tr>

  <tr>
    <th scope="row">Tag Prefixed With</th>
    <td>Tag foo is prefixed with bar.</td>
    <td><pre><code class="language-hql">&lt;foo&gt; ^ &lt;bar&gt;</code></pre></td>
    <td><pre><code class="language-json">["^", &lt;foo&gt;, &lt;bar&gt;]</code></pre></td>
  </tr>

  <tr>
    <th scope="row">Tag Not Prefixed With</th>
    <td>Tag foo is not prefixed with bar.</td>
    <td><pre><code class="language-hql">&lt;foo&gt; !^ &lt;bar&gt;</code></pre></td>
    <td><pre><code class="language-json">["not", ["^", &lt;foo&gt;, &lt;bar&gt;]]</code></pre></td>
  </tr>

  <tr>
    <th scope="row">True</th>
    <td>True.</td>
    <td><pre><code class="language-hql">true</code></pre></td>
    <td><pre><code class="language-json">true</code></pre></td>
  </tr>

  <tr>
    <th scope="row">False</th>
    <td>False.</td>
    <td><pre><code class="language-hql">false</code></pre></td>
    <td><pre><code class="language-json">false</code></pre></td>
  </tr>

  <tr>
    <th scope="row">RegEx On Tag</th>
    <td>Regular expressions on tags.</td>
    <td><pre><code class="language-hql">&lt;a&gt; ~ &lt;b&gt;</code></pre></td>
    <td><pre><code class="language-json">["~", &lt;a&gt;, &lt;b&gt;]</code></pre></td>
  </tr>

  <tr>
    <th scope="row">Custom Filters</th>
    <td>Details in the next section.</td>
    <td><pre><code class="language-hql">q &lt;a&gt;</code></pre></td>
    <td><pre><code class="language-json">["q", &lt;a&gt;]</code></pre></td>
  </tr>

</table>
</div>

<h4 id="customfilters">
  Custom Filters
  <a class="link-to" href="#customfilters"><span class="glyphicon glyphicon-link"></span></a>
</h4>

<p>
  When you need to select series using a more complex syntax than that available through the basic tag suggestions, you can create custom filters. These can take advantage of the HQL format of the above filters.
</p>

<h5>Some examples:</h5>

<p>
Suppose you want to look at unsuccessful incoming requests. How do we select the unsuccessful ones from all the other incoming requests? One way is to create a custom filter like this:
</p>
<code class="language-json">result in ["fail", "drop"]</code>

<p>
Suppose you want to exclude a single host from your query. The way to accomplish this is to use a custom filter like this:
</p>
<code class="language-json">host != "myhost.example.net"</code>

<p>
If you want to exclude more than one host, you could use a not in expression to exclude multiple values for the same tag, like this:
</p>
<code class="language-json">host not in ["myhost1.example.net", "myhost2.example.net"]</code>


<h3 id="hql">
  The HQL Language
  <a class="link-to" href="#hql"><span class="glyphicon glyphicon-link"></span></a>
</h3>

<h4>Primitives</h4>

<div class="table-responsive">
<table class="table table-bordered">
  <tr>
    <th scope="row">Simple String</th>
    <td>
      String made up of a limited set of characters for convenience.
      It must not match a parsed keyword.
    </td>
    <td>
      <pre><code class="language-hql">hello.world</code></pre>
    </td>
  </tr>

  <tr>
    <th scope="row">Quoted String</th>
    <td>String which is quoted to support <em>any</em> set of characters. Supports the same <a href="https://docs.oracle.com/javase/tutorial/java/data/characters.html">escape sequences as Java</a>.</td>
    <td>
      <pre><code class="language-hql">"hello world"</code></pre>
    </td>
  </tr>
</table>
</div>

<h4>Arithmetic Expressions</h4>

<div class="table-responsive">
<table class="table table-bordered">
  <tr>
    <th scope="row">Addition</th>
    <td><pre><code class="language-hql">&lt;a&gt; + &lt;b&gt;</code></pre></td>
    <td>
      Valid operands are:
      <code class="language-hql">&lt;string&gt; + &lt;string&gt;</code>,
      and <code class="language-hql">&lt;number&gt; + &lt;number&gt;</code>.
    </td>
  </tr>

  <tr>
    <th scope="row">Subtraction</th>
    <td><pre><code class="language-hql">&lt;a&gt; - &lt;b&gt;</code></pre></td>
    <td>
      Valid operands are:
      <code class="language-hql">&lt;number&gt; - &lt;number&gt;</code>.
    </td>
  </tr>
</table>
</div>

<h4 id="special-variables">
  Special Variables
  <a class="link-to" href="#special-variables"><span class="glyphicon glyphicon-link"></span></a>
</h4>

<div class="table-responsive">
<table class="table table-bordered">
  <tr>
    <td><code class="language-hql">$key</code></td>
    <td>Can be used in most places a tag is expected. It indicates that the given expression should match the special field <em>key</em> instead of a tag.</td>
  </tr>

  <tr>
    <td><code class="language-hql">$now</code></td>
    <td>Expands to the current (server-side) timestamp in milliseconds.</td>
  </tr>
</table>
</div>

<h4 id="duration">
  Durations
  <a class="link-to" href="#duration"><span class="glyphicon glyphicon-link"></span></a>
</h4>

<p>
  Durations are represented as a numeric component with a suffix, like <code class="language-hql">3H</code>
</p>

<p>
  Valid suffixes are:
</p>

<ul>
  <li><b>ms</b> - for milliseconds</li>
  <li><b>s</b> - for seconds</li>
  <li><b>m</b> - for minutes</li>
  <li><b>H</b> - for hours</li>
  <li><b>d</b> - for days</li>
  <li><b>w</b> - for weeks</li>
</ul>

<p>
  Durations support arithmetic expressions with each other and numbers.
</p>

<p>
  The following is a valid expression:
</p>

<pre><code class="language-hql">
$now - (1d + 1H)
</code></pre>
